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■ Description 

A Method and Apparatus for Rate Distortion Optimization Based Rate Control 
BACKGROUND OF THE INVENTION: 



Advanced video coding techniques are important for multimedia storage and transmission. For 
this reason, many video coding standards have been standardized. H.264 is the latest video coding 
standard. H.264/AVC standard jointly developed by ISO and ITU-T — Joint Video Team (JVT), 
also known as MPEG-4 Part 10 and H.264 in the H.26x serial standards, has substantially 
outperformed the previous video coding standards by utilizing a variety of temporal and spatial 
predictions. Rate control is an important technique although it does not belong to the normative part 
in video coding standards. However, without rate control any video coding scheme would be 
practically useless in many applications because the client buffer may often under-flow and 
over-flow when a channel used to deliver the compressed stream is of constant bandwidth. 
Therefore, every video coding standard has its own rate control technique, for example, TM5 for 
MPEG-2 and TMN8 for H.263. 

RDO is one of important video coding techniques. It is used to select optimal motion vectors an 
optimal coding mode for every macroblock. Yet the RDO used in H.264 test model makes it difficult 
to adopt the existing rate control techniques. Because rate control usually requires a pre-determined 
set of motion vectors and coding modes to select the quantization parameter, whereas RDO requires 
a pre-deterrnined quantization parameter to select motion vectors and coding modes. On the other 
hand, as the complexity ratio between coded frame, the bit allocation model and adaptive 
quantization scheme should also be improved. The invention is a method and apparatus for rate 
distortion optimization based rate control. The invention can be used for video streaming, 
transmission, and storage coding. 

SUMMARY OF THE INVENTION: 

The invention is to provide a method and apparatus of rate control for a video encoder, in 
which rate distortion optimization technique is used to improve coding efficiency. 

As shown in Figure 2, a rate distortion optimization based rate control implementation 
includes following modules: JVT processing module, rate distortion optimization based macroblock 
mode selection module, virtual buffer, and global complexity estimation module. 

JVT processing module receives the input frame data, and it is connected with RDO mode 
selection module, virtual buffer module and global complexity estimation module; 

RDO mode selection module is connected with virtual buffer and global complexity 
estimation module. It receives the input signal from JVT processing module, and processes it based 
on the virtual buffer module and global complexity module status. In the last, the output signal is 
sent back to JVT processing module, JVT module will output the final coded macroblock. 

Before coding a GOP, does bit allocation for the pictures in the GOP with the average picture 
size; The average picture size is calculated as: 

R/F = R+F, here, R is the target bit rate. F is the picture rate. R/F is the average picture size. 

The bit allocation adjustment in the coded GOP is shown as follows: 
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here, 7}, 7J, and 7^ is the bits allocated to the I, P or B frame respectively. N n N p and N b is the 
remained none coded I, P or B frames in the GOP respectively. X { > X p and X b is the global complexity 
estimation for the I, P or B frame respectively and is defined as the multiplier between coded bits 
and average quantization parameter for the frame. 

bitj-ate is the target bit rate, picture jrate is the frame rate. 

K p and K b are constants. K p9 K b means the complexity ration between P, B frame and I frame 
respectively 

R is the remained bits for the GOP, and after coding a picture is updated as follows: 



S iiP>b is the coded bits for the current frame. 
Before coding a GOP, the remaining bits for the current GOP is initialized as follows: 



here, R is the remained bits for the current GOP. 
Af is the number of frames in current GOP. 
G is the number of bits for a GOP. 

R prev is the remained bits for the previous GOP. For the first GOP, R prev =0. 
Xj, X p and X b are initialized as: 
Xc=2L*bit_rate 
X p =b xbitj-ate 
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X b =c*bit_rate 
here a, b and c are constants. 
bitjrate is the target bitrate. 

Does the mode selection while using the quantization parameter of previous macroblock as a 
prediction value for the current macroblock. The mode mmimizes the following expression is 
selected as the initial coding mode for the current macroblock: 

D(s,c,MODE | QP) + Z MODE R(s,c,MODE \ QP) 

here, s is the luma value of the original macroblock. c is the luma value of the reconstructed 
macroblock. \mode is the lagrangian constant. 

For I/P frame, JL MODE = 0.85 x 2*^ ; 

For B frame, X MODE =4x0.85x2 /3 . 

D(s,c,MODE\QP) is used to evaluate the distortion of the current macroblock after it is 

coded. 

R(s,c,MODE\QP) is the bits used to code the macroblock with mode MODE. 

QP is the quantization parameter for current macroblock. 
for motion estimation in P or B frame, the motion vectors rnmimizes following expression are 
selected as the motion vectors for the current macroblock: 

J ( m >A M0TJ0N ) = SA(T)D(s,c(m)) + X MOTION R(m - p) 

here, D(s,c(m)) is used to evaluate the distortion from motion compensation. 

SA (T) D is sum of the absolute difference (or after Hadmard transform) for the macroblock. 

R(m-p) is the bits used to code the motion vector. 

s is the luma value of the current macroblock in the original frame. 

c is the luma value in reference picture. 

m is the motion vector. 

p is the predicted motion vector. 

^•motion is the lagrangian constant and A MOTJON = *jA MODE . 

^■mode is the lagrangian constant. 

After the first rate distortion mode selection, the output of RDO mode selection module is 



sent to JVT processing module. A new quantization parameter will be calculated by the JVT 
processing module. The quantization parameter is adjusted according to macroblock activity. 

After first rate distortion mode selection, the sum of the absolute difference is used as the 
macroblock activity estimation. The macroblock activity is calculated as: 

[ixacttj+avg act 
m i j act j + (2 x avg _ act) 

here, i is the horizontal position of the pixel in the current macroblock.y is the vertical position of the 
pixel in the current macroblock. N_act m is the activity of the current macroblock. s(ij) is the luma 
value of the original pixel(i,y), c(ij) is the prediction value of pixel(i,7'). avg_act is the average act m 
in the previous coded picture which is coded with the same type as current picture. act m is the sum of 
the absolute difference after motion compensation or intra prediction. 

When coding the first frame, the virtual buffer occupancy is initialized with: 

rfj=10xr/31 
d 0= K b* d 0 

here r is the virtual buffer size; do, d 0 p \ and d 0 b is the initial virtual buffer occupancy for i, p, 
or b frame. K p is the complexity ration between I, P frame; K b is the complexity ratio between I,B 
frame. 

The RDO based rate control also includes a second RDO mode selection, after quantization 
parameter decision for the current macroblock. That is to say, the decided quantization parameter 
for the current macroblock will be used to RDO mode selection again. The mode which mirumizes 
the following expression will be selected as coding mode for the current macroblock: 

here, s is the luma value of the original macroblock. c is the luma value of the reconstructed 
macroblock. X M0DE is the lagrangian constant. 

For I/P frame, X MODE = 0.85 x 2 A ; 
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For B frame, X MODE = 4 x 0.85 x 2 /3 . 

D(s,c,MODE\QP) is used to evaluate the distortion of the current macroblock after it is 



coded. 

R(s,c,MODE\QP) is the bits used to code the macroblock with mode MODE, 
QP is the quantization parameter for current macroblock. 
Quantization parameter from JVT processing module is sent back to JVT processing module, the 

macroblock is coded by JVT processing module and output. 

Based on above modules, the drawbacks of traditional rate control schemes are removed. As 
RDO and rate control are considered together, the RDO based video coding can reach accurate 
target bitrate control while with good performance. 



